I struggled with the same problem.
My problem was caused by sharing WiFi of iMac.
The most obvious solution seems to be to disable the WiFi of the mac running XCode and use the wired network.
XCode : 12.3
macOS: 11.0.1
iMac : 2019
Topic:
Developer Tools & Services
SubTopic:
Xcode
Tags: